…-runner skill phase

Phase 1 status (shipped in #233 minus the trial-case port), and a new
Phase 5: a Claude Code skill packaging the Khan/webapp#40678 seeded
live-PR trial choreography (isolated arm PRs with distinct workflow
names, run collection, defect-table scoring, corpus-case export,
cleanup). Seeds and ground truth stay human-authored.

…record the judge-direction lesson

All three follow-ups shipped: agent-failure reasons and the per-row
stability footer on #236, lens routing for live cases on #233/#235.
Records the acceptance pair's most instructive number: judge quality
rose on the deliberately regressed arm (fewer, surer comments score
better), so recall against labeled specs is the load-bearing metric.

Phase 2 of the live A/B eval plan (#232), complete, stacked on the Phase 1 corpus format (#233): everything needed to run the real model sub-agents over a live-enabled corpus case.

## 2a: `eval/agent-extract.ts`

Parses `review.md`'s `## agent:` sections into `{name, description, model, prompt}`. String in, no filesystem access, because the baseline arm of an A/B extracts the merge-base `review.md` via `git show`. Parsing is strict: missing/unterminated frontmatter, a name mismatch, a missing model, or an empty body throws with every problem listed, since a silently dropped agent would skew an eval arm without failing it. An integration test extracts the real `review.md` (21 agents) and pins the default roster's reference to the production staging root so a rename fails a test instead of silently staging nothing.

## 2b: `eval/live-stage.ts`

Materializes the production `/tmp/gh-aw/review/` layout for one case, per `review.md` Step 1's staging contract: `pr-context.json` (synthetic identity fields), `full.diff`/`pr.diff`/`full-stripped.diff`, `files.json` + `review-files.json` with the `hasPatch` cross-check, `provenance.json`, `routing.json` from the deterministic router, `out/`, and the post-change checkout from the case's tree. `rewriteAgentPrompt` swaps the production staging root across a prompt.

## 2c: `eval/live-producer.ts` + `eval/live-runner.ts`

`produceLive` dispatches the live roster (default finders plus the router's `lensesToSpawn`; no pattern-triage or thread-reconciler in eval) behind an injected `LiveAgentRunner` seam, the same pattern as `judge.ts`'s injected model, so all its logic is stub-tested with zero model calls. It maps the three sub-agent output contracts into exactly the shapes the deterministic runner consumes (`RecordedFinding[]` for `produceFindings`, `CaseVerification[]` for the validation replay): label-shape findings get the code-assigned lens (`correctness`, or `conventions` for the skill-auditor so `labelForFinding` reproduces the best-practice label variants) and the production 0.7 confidence default; lens findings validate against the schema as-is; the claim-validator's `{claims: [...]}` output becomes verifications. It stages `claims.json`, resolves `{{#runtime-import}}` directives from the case tree (a case opts into a skills index by carrying the file), retries once on malformed output with the rejection fed back, keeps partial results when an agent fails twice, prefixes colliding finding ids, and accounts per-agent cost/turns/wall-clock.

`live-runner.ts` is the only module touching a real model runtime: one Agent SDK `query()` per dispatch, Read/Grep/Glob only, cwd pinned to the staged checkout, the agent's pinned model, hard turn and wall-clock caps, plus the CLI smoke entry: `pnpm dlx tsx workflows/review/eval/live-runner.ts --case <id>` (requires `ANTHROPIC_API_KEY`). Adds `@anthropic-ai/claude-agent-sdk` as a dev dependency.

Documented deviations from production (module doc): no pattern-triage pass, runtime imports resolve from the case tree or a fixed note, and the investigation-cap CLI is not staged (its own denied-budget fallback applies).

## Test plan:

- `pnpm run test --run`: 680 tests green (21 new across the three test files: extraction fixtures + real-review.md integration, staging via memfs, producer via a scripted stub runner covering happy path, retry, double-failure, empty-findings skip, id collisions, missing-agent error, and runtime-import resolution).
- `pnpm run typecheck` and eslint clean.
- The CLI smoke run against a live case needs `ANTHROPIC_API_KEY`, which this environment does not have; it is the first thing to run when testing this PR (any of the 13 live case ids, e.g. `--case incident-money-rounding`). Expected cost: roughly $0.50 to $2 per case with the default roster.


## Next steps (human)

1. ~~Key-bearing validation~~ DONE, superseded by the phase 4 acceptance runs, which exercised this producer end to end with a real key over 7 live cases x 2 arms x 2 PRs (runs [29052332224](https://github.com/Khan/actions/actions/runs/29052332224) and [29052334404](https://github.com/Khan/actions/actions/runs/29052334404)); measured ~$0.55-0.72 per case per arm, inside the plan's expected envelope. The CLI smoke remains available for local iteration.
2. Sanity-check the SDK runner's settings (`live-runner.ts`): `permissionMode: "bypassPermissions"` with tools restricted to Read/Grep/Glob and cwd pinned to the staged checkout. Confirm you are comfortable with that posture for CI.
3. Review the documented deviations from production in `live-producer.ts`'s module doc (no pattern-triage, runtime-import resolution, investigation-cap absence); each is a measurement caveat for the A/B.
4. New dev dependency: `@anthropic-ai/claude-agent-sdk`. Confirm it clears whatever dependency review applies to this repo.

Author: jwbron

Reviewers: jeresig, github-actions[bot], jaredly

Required Reviewers:

Approved By: jeresig, github-actions[bot]

Checks: ✅ 9 checks were successful

Pull Request URL: #234

Ports the seeded-defect trial from Khan/webapp#40678 into the review-workflow eval corpus as three live-enabled cases, stacked on #233 (the live-enabled corpus format).

## Sanitization constraint

Khan/actions is public and Khan/webapp is private, so these are sanitized structural rewrites, not copies: fresh Go code around a generic "notes retention" feature (per-user stored notes, a background prune, deletion on account erasure) that reproduces the trial's defect mechanisms with generic naming. No webapp code, file paths, or identifiers appear anywhere in the cases.

## The three cases

### `trial-retention-deletion` (incident-repro, tags incident/trial/live)

The trial's deletion-path seeds, five recorded findings across `erasure.go`, `store.go`, and a new test file:

- `erasure-flag-gated-deletion` (blocking): the account-erasure deletion is gated behind a rollout flag, so flag-off silently skips the compliance deletion (trial seed 1).
- `erasure-default-limit-one` (blocking): the deletion query leaves `Limit` unset and the store's documented zero-value default is 1, so at most one note is ever deleted (seed 2).
- `erasure-ignores-delete-helper` (advisory): the new loop reimplements, worse, a paging batch-delete helper that already exists in the same file (seed 3).
- `erasure-env-widening` + `store-env-widening` (advisory, one per file): the request-env interface is widened in two files solely for the buggy flag gate (seed 4).
- `erasure-prune-error-swallowed` (blocking): `_ = PruneUserNotes(...)` discards the returned error; the prose cites the error-handling skill (seed 8).

Expected: REQUEST_CHANGES, all six ids must-catch, 6 posted comments.

### `trial-retention-prune-tests` (incident-repro, tags incident/trial/live)

The prune-and-tests seeds in a new `prune.go` + `prune_test.go` (both added files):

- `prune-off-by-one-cap` (advisory, the trial's calibrated severity): `notes[maxRetainedNotes-1:]` retains 199 where the documented cap is 200 (seed 5).
- `prune-test-vacuous-cap` (blocking): the cap test seeds 5 notes, far below the cap, so the early return makes it pass even for a no-op prune (seed 6).
- `prune-suite-flag-mock` (blocking): `TestMain` forces the rollout flag on for the whole suite, so the flag-off path is never exercised (seed 7).
- `prune-full-entity-fetch` (advisory): the prune fetches up to 5000 full entities but reads only ids; `Query.KeysOnly` exists for exactly this (seed 9).

Expected: REQUEST_CHANGES, all four ids must-catch, 4 posted comments.

### `trial-batch-delete-wrapper` (clean, tags clean/trial/live)

The trial's deliberate non-defect (seed 10) as a must-not-flag trap: the changed `purge.go` collects every note key and issues one `DeleteMulti` call that looks like it exceeds the datastore's 500-entity per-call cap. The tree also carries the unchanged wrapper source (`internal/datastore/client.go`) whose `DeleteMulti` visibly chunks into 500-key batches, so a live reviewer can discover the truth by reading it. The case records the wrong blocking claim, neutralizes it with a `refuted` validation entry, and pins `expected` to APPROVE with zero posted comments; `live.mustNotFlagSpecs` describes the trap so a live run that claims the cap is scored as a false block.

## Case mechanics

- Each case carries a git-style unified diff computed from authored pre/post file contents (difflib-generated, added test files staged against `/dev/null` with a new-file mode line), the post-change tree next to `case.json`, `live.prContext`, and per-defect `mustCatchSpecs`/`mustNotFlagSpecs` with line windows of about +/-5 around the defect and 2-4 mechanism alternates.
- Every recorded line-anchored finding anchors on an ADDED line of the case diff; the generator asserted this against the diff, so the change-provenance gate keeps every finding.
- None of the cases are tagged `smoke`, keeping the per-PR smoke gate fast; both incident-repro cases feed must-catch recall in `suite.test.ts` automatically.
- Includes a `review` patch changeset.

## Test plan:

- `pnpm run test --run`: all 21 files / 662 tests green, including the corpus loader's live-block and tree validation over the new cases and `checkExpectation` on each case's expected block.
- `pnpm run typecheck`: clean (no TypeScript touched).
- Provenance-gate anchors verified mechanically at generation time: every finding's anchor line was asserted to be an added line of its file's diff.


## Next steps (human)

1. Sanitization check, the load-bearing review here: someone who knows the webapp trial code should confirm no webapp code, paths, or identifiers leaked into these three cases (Khan/actions is public; the cases must be structural rewrites only).
2. Severity calls to ratify: seed 7 (suite-wide flag mock) was authored as blocking by judgment call; seed 5 (off-by-one) is advisory per the trial's calibration. Adjust if you disagree; `expected` blocks and specs move together.
3. Rebase and merge after #233; the deterministic suite gates the rest.

---

## Update 2026-07-09: two misses promoted to smoke

`trial-dedup-composite-key` and `trial-erasure-suite-flag-mock` (the v1.4.0 re-run misses) now carry the `smoke` tag, so every per-PR A/B watches the two defects the current production prompt demonstrably missed (~$1/run for the pair). The other trial cases deliberately stay behind the `full-eval` label.

Author: jwbron

Reviewers: jeresig, github-actions[bot], somewhatabstract

Required Reviewers:

Approved By: jeresig, github-actions[bot]

Checks: ✅ 9 checks were successful

Pull Request URL: #235

Phase 3 of the live A/B eval plan (#232), stacked on the Phase 2 producer (#234): score live runs against labeled ground truth and diff two review.md versions arm to arm.

## 3a: `eval/live-match.ts`

Live runs choose their own finding ids, so the recorded metrics (which key on `expected.mustCatch` ids) cannot score them. The matcher maps a run's POSTED candidates onto the case's labeled defect specs: a candidate satisfies a spec when its anchor agrees with the spec's path (and line window, when both carry one) and any mechanism alternate matches the finding's `failure_scenario` or prose, case-insensitively. Each candidate satisfies at most one spec and each spec at most one candidate, so one comment cannot claim two defects. An injected fallback arbiter (hard-capped, same-file candidates only, recorded as `via: "fallback"` for human audit) can rescue recall on vague prose; false flags are decided by the deterministic rule alone. `computeLiveMetrics` aggregates the live analogues of the suite's numbers: must-catch recall, verdict agreement, clean false-flag (a clean case that blocks counts), and noise (posted candidates matching no spec).

## 3b: `eval/live-ab.ts`

The arm orchestrator and CLI. Baseline `review.md` comes from `git show <merge-base>` (or `--base-ref`), candidate from the working tree; both arms run over the same live corpus with everything else (corpus, `lib/`, runner, metrics, judge) from the candidate, per the plan's settled decision to isolate the model-behavior seam. Each arm runs under half the `--max-usd` budget (default 40 total) with sticky exhaustion: once spend plus the running per-case average crosses the cap, the remaining cases are recorded as SKIPPED and the report still emits (dying at a cap with nothing posted is the failure mode the plan forbids). Each live result replays through the deterministic pipeline; `runner.ts` gains an optional `RunOptions.validation` override so the live validator's output replaces the recorded block. Spec-level regressions ("baseline caught, candidate missed" and the reverse) are diffed only over cases both arms actually scored, so a budget skip is never reported as a regression.

Judge scoring reuses the pinned judge for quality aggregates only (judge-vs-ground-truth disagreement keys on recorded ids, which live arms do not use); the fetch model moves from `live-judge.ts` into shared `eval/judge-live-model.ts`.

Output: `out/live-ab-report.json` plus a markdown table (metrics deltas, judge quality delta, cost, wall clock, regressions, skips, agent failures) printed and appended to `GITHUB_STEP_SUMMARY`. Report-only, with one exception per the playbook's standing rule: the candidate arm failing any adversarial-injection case (wrong verdict or missed spec) exits non-zero.

## Test plan:

- `pnpm run test --run`: 696 tests green (16 new: matcher rules incl. window overlap, one-candidate-one-spec, malformed-regex fallback-to-literal, capped fallback audit trail; runArm budget semantics incl. the sticky-stop case that caught a real bug in review; regression diffing over shared cases; report rendering both gate outcomes).
- `pnpm run typecheck` and eslint clean.
- The end-to-end CLI (`pnpm dlx tsx workflows/review/eval/live-ab.ts --max-usd 10 --cases incident-money-rounding,clean-no-findings`) needs `ANTHROPIC_API_KEY`; this environment has none, so that is the first thing to exercise when testing. On an unchanged review.md it prints the identical-arms note and near-zero deltas.


## Next steps (human)

1. ~~End-to-end A/B validation~~ DONE, superseded by the phase 4 acceptance runs (linked on #237): the control arm held recall and verdict agreement flat at +0% and the weakened arm showed -17% recall with the lost spec named, which is also the first empirical read on matcher quality (7 cases matched; one systematic miss traced to lens routing, fixed on #233, not to the matcher). Two acceptance-driven additions landed here since: `perCase.failedAgents` entries now carry `<agent>: <reason>` (a claim-validator flake was undiagnosable from the report alone; the flake itself is still to be root-caused on a future run), and every report closes with a per-row stability footer (judge quality rose on the deliberately regressed arm because fewer, surer comments each score better, so the footer warns against reading it alone).
2. Review the two scoring caveats: judge output is quality-aggregates only (disagreement semantics key on recorded ids), and the fallback arbiter is wired but not yet given a live implementation (matches would be marked `via: "fallback"` for audit when one is added).
3. Confirm the budget semantics (half of `--max-usd` per arm, sticky skip-and-report) match how you want cost bounded.

---

## Update 2026-07-09: eval-tuning pass (three riders)

Three instrument fixes from the [eval-tuning memo](https://claude.ai/code/artifact/2995f0b5-8840-4d03-96e3-2518e2f0f75a), landed here so the whole stack inherits them:

1. **Pre-flight identity short-circuit** (memo item 1, hoisted down from #251 with identical text so that branch merges cleanly): byte-identical review.md in both arms posts a no-reviewable-delta report and runs nothing; `--force-arms` preserved for wobble controls. Corpus-only and plumbing-only PRs in this stack now pay $0 per push.
2. **Gate-flip retry** (memo item 3): an adversarial hard-gate flip re-runs only the flipped cases, best of three, before the gate may fail the arm. Retried runs never replace the original in the metrics; they only decide whether the flip was a run-to-run flake (the #244/#245 incident shape). Retry spend is recorded in the report.
3. **Drop-bucket taxonomy** (memo item 4): every missed must-catch spec is classified true-miss vs found-but-dropped (provenance/scope/validation) by re-matching against the dropped candidates with the line window relaxed. Lost regressions are annotated with their class and the table gains a found-but-dropped count row.

Plus **judge economics**: the judge pin moves from Opus 4.8 to `claude-haiku-4-5-20251001` (the judge grades prose only; every load-bearing metric is deterministic, and the acceptance runs showed judge quality is not single-run-stable on any model, so it is priced accordingly; supersedes operator direction 4), and the judge seam gains retry with backoff so a transient 500 no longer wastes a whole scoring pass.

Author: jwbron

Reviewers: jeresig, github-actions[bot], jwbron, jaredly

Required Reviewers:

Approved By: jeresig, github-actions[bot]

Checks: ✅ 9 checks were successful

Pull Request URL: #236

Phase 5 of the live A/B eval plan (#232), independent of the eval-code stack (#233-#237): the `review-trial` Claude Code skill, packaging the Khan/webapp#40678 seeded-defect trial choreography so an operator can run one in an afternoon.

## What the skill does

Given a consumer repo, a seeded branch, a **human-authored** defect table, an arms list, and budget approval (projected up front from the trial's measured $7-10 per workflow-arm run, confirmed before any PR is created), it:

1. Creates one isolated PR per arm from the seeded branch, with per-arm trigger recipes (repo-default reviewer, pinned workflow ref, hosted `@claude review`) and two hard rules: distinct workflow names (same-named gh-aw workflows share a per-PR concurrency group and cancel each other; this silently ate a run in the original trial) and exactly one reviewer per PR (suppress the default via `skip-ai-review` on non-default arms).
2. Collects each run's review, artifacts (tolerating the known gh-aw staging-path artifact annotation), billed credits, and wall clock.
3. Optionally drives the lifecycle protocol (push 2 with fixes plus fresh seeds, push 3 all-fixed) identically across arms, scoring each push separately.
4. Scores defect by defect with the deterministic rule mirroring `eval/live-match.ts` (path + window + mechanism), manual judgment for leftovers marked as judged in the report, traps counted as correct suppression, and faithful reporting whichever arm it favors; output in the #40678 report shape.
5. Exports live-enabled corpus case skeletons, with an explicit sanitization gate for private-repo content landing in this public repo (the #235 lesson: structural rewrites, never copied code).
6. Cleans up: PRs closed with the report linked, trial branches deleted, no temporary workflow left behind.

What stays human is stated at the top of the skill: seeds and ground truth are operator-authored, always; the skill refuses to improvise a defect table.

## `.gitignore` change

`.claude` was ignored wholesale, which would have kept project skills untracked. Narrowed to `.claude/*` with a `!.claude/skills/` carve-out; verified `settings.json` and `worktrees/` remain ignored.

## Test plan:

Docs/tooling only (empty changeset); no code paths. `git check-ignore` verified the carve-out (SKILL.md tracked, `settings.json` and `worktrees` still ignored). The real acceptance is the plan's: re-running the #40678 trial via this skill reproduces its table in an afternoon, which needs a consumer-repo trial to exercise.


## Next steps (human)

1. Review the `.gitignore` change deliberately: `.claude` wholesale becomes `.claude/*` + `!.claude/skills/`. Verified locally that `settings.json` and `worktrees/` stay ignored, but this is a policy change about what agent tooling gets committed to the repo.
2. The skill's acceptance is operational, not CI: next time a trial is warranted (an architecture-bet change like the deterministic orchestrator, or pre-graduation ground-truthing), run it via this skill and check it reproduces the #40678 report shape in an afternoon. There is no cheaper honest validation.
3. Skim the per-arm trigger recipes against current consumer-repo reality (the `skip-ai-review` label convention, ready-for-review semantics); those conventions live outside this repo and can drift.

Author: jwbron

Reviewers: jeresig, github-actions[bot], somewhatabstract

Required Reviewers:

Approved By: jeresig, github-actions[bot]

Checks: ✅ 10 checks were successful

Pull Request URL: #238

Adds `workflows/review/eval/live-ab-plan.md`: a phased, agent-executable plan for real-model evals of review-bot changes, making the round-two doc's eval playbook executable for its behavior-shifting and cost-affecting change classes (today only the deterministic replay exists, which is structurally blind to prompt and model changes).

The phases:

1. **Live-enabled corpus cases**: real diffs, post-change file trees, PR context, and labeled defect specs (live runs choose their own finding ids, so ground truth matches on anchor window + mechanism, not id). Implementation is up in a separate PR.
2. **Live producer**: run the actual sub-agent roster from `review.md` over a case via the Agent SDK (Read/Grep/Glob only, staged checkout, per-agent model pinning, live claim-validator, cost capture), behind the `produceFindings` seam `runner.ts` already ships.
3. **A/B runner + delta report**: same-run arms (merge-base `review.md` vs working tree), five-metric deltas, judge quality deltas, per-case cost, adversarial hard gate outright.
4. **CI wiring**: `review-eval-ab.yml` on `paths: [workflows/review/**]`, sticky PR comment, smoke-live subset by default, `full-eval` label escalation.

Settled decisions recorded in the doc: same-run A/B over stored baselines; candidate corpus + lib for both arms (isolate the model seam); bounded agentic sub-agents, not one-shot completions; report-only except the adversarial gate; budget caps that still emit a partial report. Also records how this instrument relates to seeded live-PR trials like Khan/webapp#40678 (complementary layers, different cadences).

## Test plan:

Docs only (plus an empty changeset). `pnpm run test --run` stays green.


## Next steps (human)

1. Review the settled design decisions section; they bind the whole stack (#233-#238), so disagreement is cheapest to voice here.
2. Two decisions the plan deliberately leaves to humans, no code blocked on them: ratify the gate thresholds that would turn the A/B report into a merge gate (until then everything is report-only except the adversarial rule), and decide whether `pattern-triage` joins the live roster.
3. This PR is docs plus an empty changeset; it can merge independently of the stack whenever the plan reads right. All five phases have landed and the doc now records the phase 4 acceptance results (both arms passed; the plan's acceptance section names the runs and the judge-direction lesson), so the remaining human items are exactly the two decisions in step 2 plus the consumer-repo trial that accepts #238.

Author: jwbron

Reviewers: jeresig, github-actions[bot], jwbron

Required Reviewers:

Approved By: jeresig, github-actions[bot]

Checks: ✅ 9 checks were successful

Pull Request URL: #232

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* review: add the review-trial skill (live A/B phase 5) (#238)

Phase 5 of the live A/B eval plan (#232), independent of the eval-code stack (#233-#237): the `review-trial` Claude Code skill, packaging the Khan/webapp#40678 seeded-defect trial choreography so an operator can run one in an afternoon.

## What the skill does

Given a consumer repo, a seeded branch, a **human-authored** defect table, an arms list, and budget approval (projected up front from the trial's measured $7-10 per workflow-arm run, confirmed before any PR is created), it:

1. Creates one isolated PR per arm from the seeded branch, with per-arm trigger recipes (repo-default reviewer, pinned workflow ref, hosted `@claude review`) and two hard rules: distinct workflow names (same-named gh-aw workflows share a per-PR concurrency group and cancel each other; this silently ate a run in the original trial) and exactly one reviewer per PR (suppress the default via `skip-ai-review` on non-default arms).
2. Collects each run's review, artifacts (tolerating the known gh-aw staging-path artifact annotation), billed credits, and wall clock.
3. Optionally drives the lifecycle protocol (push 2 with fixes plus fresh seeds, push 3 all-fixed) identically across arms, scoring each push separately.
4. Scores defect by defect with the deterministic rule mirroring `eval/live-match.ts` (path + window + mechanism), manual judgment for leftovers marked as judged in the report, traps counted as correct suppression, and faithful reporting whichever arm it favors; output in the #40678 report shape.
5. Exports live-enabled corpus case skeletons, with an explicit sanitization gate for private-repo content landing in this public repo (the #235 lesson: structural rewrites, never copied code).
6. Cleans up: PRs closed with the report linked, trial branches deleted, no temporary workflow left behind.

What stays human is stated at the top of the skill: seeds and ground truth are operator-authored, always; the skill refuses to improvise a defect table.

## `.gitignore` change

`.claude` was ignored wholesale, which would have kept project skills untracked. Narrowed to `.claude/*` with a `!.claude/skills/` carve-out; verified `settings.json` and `worktrees/` remain ignored.

## Test plan:

Docs/tooling only (empty changeset); no code paths. `git check-ignore` verified the carve-out (SKILL.md tracked, `settings.json` and `worktrees` still ignored). The real acceptance is the plan's: re-running the #40678 trial via this skill reproduces its table in an afternoon, which needs a consumer-repo trial to exercise.


## Next steps (human)

1. Review the `.gitignore` change deliberately: `.claude` wholesale becomes `.claude/*` + `!.claude/skills/`. Verified locally that `settings.json` and `worktrees/` stay ignored, but this is a policy change about what agent tooling gets committed to the repo.
2. The skill's acceptance is operational, not CI: next time a trial is warranted (an architecture-bet change like the deterministic orchestrator, or pre-graduation ground-truthing), run it via this skill and check it reproduces the #40678 report shape in an afternoon. There is no cheaper honest validation.
3. Skim the per-arm trigger recipes against current consumer-repo reality (the `skip-ai-review` label convention, ready-for-review semantics); those conventions live outside this repo and can drift.

Author: jwbron

Reviewers: jeresig, github-actions[bot], somewhatabstract

Required Reviewers:

Approved By: jeresig, github-actions[bot]

Checks: ✅ 10 checks were successful

Pull Request URL: #238

…g-relative paths) (#243)

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

…eep-list (#244)

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

)

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* [jwbron/review-out-of-lane] ci: re-fire checks against the retargeted main base

…ile (#247)

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* [jwbron/review-dispatch-tax] review: verify the disciplines extraction against real sed, not only the modeled replica

* [jwbron/review-dispatch-tax] review: drop the per-lens out-of-lane copies the main merge re-imported (the shared disciplines section carries the text)

#248)

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file

* [jwbron/review-skill-rule-quote] review: render the quoted skill rule into the comment the author reads

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* [jwbron/review-skill-rule-quote] review: answer the rule_quote prose review notes (spell out the section reference, drop the rendering detail)

* [jwbron/review-skill-rule-quote] review: keep a multi-line rule_quote inside its blockquote

* [jwbron/review-skill-rule-quote] review: restore the JSDoc continuation prefix (review feedback)

…246)

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals

The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING
(full | scoped | flip-gated | fast, default full) dials how much of the
roster a repeat review runs. lib/rereview-mode.ts owns the deterministic
half: content-hashed hunk signatures over the generated-stripped diff,
the hidden review-body fingerprint stamp (survives cache eviction,
dismiss-stale-approvals, and COMMENTED-only histories), the divergence
tripwire that re-arms full review at unreviewed share >= 0.4, and the
plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval,
sparse-PR-then-payload) ship as data in eval/lifecycle/ with a
deterministic replay harness; counters gain costByRereviewDepth so the
live A/B prices scoped against full.

The review.md orchestrator wiring lands in a follow-up commit, on top of
fold-in batch two's re-review accountability surface.

* [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer

The re-review mode dial's flip gate needs 'did the reconciler keep any
blocking thread' as a number it can read from rereview.json, not a label
judgment re-made at verdict time. Additive: the section rendering is
unchanged.

* [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator

Step 1 stages the bot's prior reviews (every state, dismissed and
comment-only included) for the stamp reader. Step 3 runs the
rereview-mode plan CLI after the provenance CLI and maps each depth to
its dispatch and staging: scoped overwrites full-stripped.diff with
scoped.diff so the full roster reads only unseen hunks; flip-gated
dispatches reconcile plus correctness over the scoped diff; fast is
reconcile-only. Step 4 gains the flip rule: a reduced-depth run may
flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0,
and in flip-gated depth a validated blocking finding posts and blocks,
vetoing the flip. Step 6 appends the hidden fingerprint stamp as the
last line of every submitted review body; Steps 7 and 8 skip on
reduced depths; Step 9 notes the stamp, not the cache, is the
tripwire's authoritative fingerprint. The executed plan is copied into
out/ so the run artifact records the depth the counters price.

* [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree

The dispatch-tax trim brought workflows/review into eslint+prettier
scope; format the mode-dial files and the two pre-existing violations
in disciplines.test.ts and finding-schema.ts so CI lints clean.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* [jwbron/rereview-mode-dial] review: fix the rereview-mode CLI usage string (review feedback)

* [jwbron/rereview-mode-dial] review: tighten the budget-table comment under the max-lines limit (the main merge pushed router.ts to 1004)

…ycle chain) (#251)

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals

The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING
(full | scoped | flip-gated | fast, default full) dials how much of the
roster a repeat review runs. lib/rereview-mode.ts owns the deterministic
half: content-hashed hunk signatures over the generated-stripped diff,
the hidden review-body fingerprint stamp (survives cache eviction,
dismiss-stale-approvals, and COMMENTED-only histories), the divergence
tripwire that re-arms full review at unreviewed share >= 0.4, and the
plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval,
sparse-PR-then-payload) ship as data in eval/lifecycle/ with a
deterministic replay harness; counters gain costByRereviewDepth so the
live A/B prices scoped against full.

The review.md orchestrator wiring lands in a follow-up commit, on top of
fold-in batch two's re-review accountability surface.

* [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer

The re-review mode dial's flip gate needs 'did the reconciler keep any
blocking thread' as a number it can read from rereview.json, not a label
judgment re-made at verdict time. Additive: the section rendering is
unchanged.

* [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator

Step 1 stages the bot's prior reviews (every state, dismissed and
comment-only included) for the stamp reader. Step 3 runs the
rereview-mode plan CLI after the provenance CLI and maps each depth to
its dispatch and staging: scoped overwrites full-stripped.diff with
scoped.diff so the full roster reads only unseen hunks; flip-gated
dispatches reconcile plus correctness over the scoped diff; fast is
reconcile-only. Step 4 gains the flip rule: a reduced-depth run may
flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0,
and in flip-gated depth a validated blocking finding posts and blocks,
vetoing the flip. Step 6 appends the hidden fingerprint stamp as the
last line of every submitted review body; Steps 7 and 8 skip on
reduced depths; Step 9 notes the stamp, not the cache, is the
tripwire's authoritative fingerprint. The executed plan is copied into
out/ so the run artifact records the depth the counters price.

* [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree

The dispatch-tax trim brought workflows/review into eslint+prettier
scope; format the mode-dial files and the two pre-existing violations
in disciplines.test.ts and finding-schema.ts so CI lints clean.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/rereview-live-corpus] review: re-review coverage for the eval corpus (open-PR cases)

A live-enabled corpus case may now carry a live.rereview block: an
open-PR snapshot with the prior review's unresolved threads (author
replies included), a stamped prior review derived from priorDiff, and
the depth plan the mode dial would compute. The live producer
dispatches the thread-reconciler on such cases at every depth and
sizes the finder roster by the plan; rereview-match.ts scores
thread-resolution accuracy, the flip-gate input, and duplicate
comments on kept threads; live-ab prices a mode via --re-review-mode
on the candidate arm. The deterministic replay learns the same rules:
computeVerdict gains keptBlockingCount (the flip rule as code) and
re-review cases render the accountability section into the planned
body. Ships golden-retention-lifecycle-1/2/3, a sanitized port of the
measured seeded live trial: planted bugs, a bad partial fix with added
bugs, then the full fix that must flip to APPROVE.

* [jwbron/rereview-live-corpus] review: the re-review mode sweep and the under-tripwire pricing case

eval/rereview-sweep.ts runs the working tree's reviewer over the
open-PR corpus cases at every dial setting and prices the dial in one
table: recall, thread resolution, flip-gate correctness, duplicates,
and dollars per mode, with the executed depth per case since the
tripwire may override the dial. golden-retention-fix-push is the
under-threshold case the cheap paths need: a one-hunk fix push
(unreviewed share 0.2) that resolves the prior blocking thread but
plants a fresh defect inside the new hunk, so scoped and flip-gated
must catch it and fast definitionally cannot. No case-format change:
the mode is a run parameter.

* [jwbron/rereview-live-corpus] review: dispatchable CI home for the re-review mode sweep

The per-mode pricing table needs somewhere visible to land: a
workflow_dispatch job (never PR-triggered; sweeps spend model dollars)
that runs rereview-sweep.ts against the chosen branch, tees the
markdown table into the job summary, and uploads the JSON report as an
artifact.

* [jwbron/rereview-live-corpus] review: manual trigger surface for every model-spending eval, and the no-delta short-circuit

From the eval-tuning memo. Labels: rereview-sweep runs the dial sweep
on a PR (sticky comment + summary + artifact), live-judge runs the
judged corpus pass on a PR head, and full-eval now fires on the
labeling event itself instead of waiting for the next push; every
workflow keeps workflow_dispatch for off-PR runs. The A/B gains the
memo's pre-flight identity short-circuit: byte-identical review.md in
both arms posts a no-reviewable-delta verdict and runs nothing, with
--force-arms preserved for deliberate wobble controls.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

…wered runs, drift watch) (#252)

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals

The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING
(full | scoped | flip-gated | fast, default full) dials how much of the
roster a repeat review runs. lib/rereview-mode.ts owns the deterministic
half: content-hashed hunk signatures over the generated-stripped diff,
the hidden review-body fingerprint stamp (survives cache eviction,
dismiss-stale-approvals, and COMMENTED-only histories), the divergence
tripwire that re-arms full review at unreviewed share >= 0.4, and the
plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval,
sparse-PR-then-payload) ship as data in eval/lifecycle/ with a
deterministic replay harness; counters gain costByRereviewDepth so the
live A/B prices scoped against full.

The review.md orchestrator wiring lands in a follow-up commit, on top of
fold-in batch two's re-review accountability surface.

* [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer

The re-review mode dial's flip gate needs 'did the reconciler keep any
blocking thread' as a number it can read from rereview.json, not a label
judgment re-made at verdict time. Additive: the section rendering is
unchanged.

* [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator

Step 1 stages the bot's prior reviews (every state, dismissed and
comment-only included) for the stamp reader. Step 3 runs the
rereview-mode plan CLI after the provenance CLI and maps each depth to
its dispatch and staging: scoped overwrites full-stripped.diff with
scoped.diff so the full roster reads only unseen hunks; flip-gated
dispatches reconcile plus correctness over the scoped diff; fast is
reconcile-only. Step 4 gains the flip rule: a reduced-depth run may
flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0,
and in flip-gated depth a validated blocking finding posts and blocks,
vetoing the flip. Step 6 appends the hidden fingerprint stamp as the
last line of every submitted review body; Steps 7 and 8 skip on
reduced depths; Step 9 notes the stamp, not the cache, is the
tripwire's authoritative fingerprint. The executed plan is copied into
out/ so the run artifact records the depth the counters price.

* [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree

The dispatch-tax trim brought workflows/review into eslint+prettier
scope; format the mode-dial files and the two pre-existing violations
in disciplines.test.ts and finding-schema.ts so CI lints clean.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/rereview-live-corpus] review: re-review coverage for the eval corpus (open-PR cases)

A live-enabled corpus case may now carry a live.rereview block: an
open-PR snapshot with the prior review's unresolved threads (author
replies included), a stamped prior review derived from priorDiff, and
the depth plan the mode dial would compute. The live producer
dispatches the thread-reconciler on such cases at every depth and
sizes the finder roster by the plan; rereview-match.ts scores
thread-resolution accuracy, the flip-gate input, and duplicate
comments on kept threads; live-ab prices a mode via --re-review-mode
on the candidate arm. The deterministic replay learns the same rules:
computeVerdict gains keptBlockingCount (the flip rule as code) and
re-review cases render the accountability section into the planned
body. Ships golden-retention-lifecycle-1/2/3, a sanitized port of the
measured seeded live trial: planted bugs, a bad partial fix with added
bugs, then the full fix that must flip to APPROVE.

* [jwbron/rereview-live-corpus] review: the re-review mode sweep and the under-tripwire pricing case

eval/rereview-sweep.ts runs the working tree's reviewer over the
open-PR corpus cases at every dial setting and prices the dial in one
table: recall, thread resolution, flip-gate correctness, duplicates,
and dollars per mode, with the executed depth per case since the
tripwire may override the dial. golden-retention-fix-push is the
under-threshold case the cheap paths need: a one-hunk fix push
(unreviewed share 0.2) that resolves the prior blocking thread but
plants a fresh defect inside the new hunk, so scoped and flip-gated
must catch it and fast definitionally cannot. No case-format change:
the mode is a run parameter.

* [jwbron/rereview-live-corpus] review: dispatchable CI home for the re-review mode sweep

The per-mode pricing table needs somewhere visible to land: a
workflow_dispatch job (never PR-triggered; sweeps spend model dollars)
that runs rereview-sweep.ts against the chosen branch, tees the
markdown table into the job summary, and uploads the JSON report as an
artifact.

* [jwbron/rereview-live-corpus] review: manual trigger surface for every model-spending eval, and the no-delta short-circuit

From the eval-tuning memo. Labels: rereview-sweep runs the dial sweep
on a PR (sticky comment + summary + artifact), live-judge runs the
judged corpus pass on a PR head, and full-eval now fires on the
labeling event itself instead of waiting for the next push; every
workflow keeps workflow_dispatch for off-PR runs. The A/B gains the
memo's pre-flight identity short-circuit: byte-identical review.md in
both arms posts a no-reviewable-delta verdict and runs nothing, with
--force-arms preserved for deliberate wobble controls.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* [jwbron/eval-measurement-tool] review: repeat aggregation, --repeats powered runs, drift watch, sql-index case fix

The eval instrument's percentage deltas sit below the one-case noise floor;
this lands the memo's measurement items. eval/aggregate.ts pools N report
artifacts into per-case pass rates with Wilson intervals and pooled rows
(reproduces the 07-10 cumulative numbers exactly); live-ab.ts --repeats runs
one powered dispatch with a strict-majority adversarial gate; the weekly
review-eval-drift.yml workflow runs full corpus x3 on main and publishes the
aggregate; identical-arm pools render noise-floor bands as data.

incident-sql-missing-index diagnosed against all 28 recorded arm-runs: the
reviewer found the missing index every single time; the 8/16 catch rate was
the spec accepting only the migration-file anchor while the reviewer anchored
the same blocking finding at the hot query (11 runs) or had it provenance-
dropped on a mis-anchor (2 runs). Specs gain altLocations; the case accepts
the query-site anchor and its residual misses now classify found-but-dropped
(the anchor-snap defect class), not recall.

* [jwbron/eval-measurement-tool] review: implement the fallback match arbiter on the pinned Haiku snapshot

Fills the MatchFallback seam in live-match.ts: when a spec stays unmatched
after the deterministic pass and a posted candidate shares its file, a
pinned claude-haiku-4-5-20251001 answers one yes/no question (same defect,
same root cause?), biased to no since a false yes inflates recall. All seam
guard rails inherited: capped per case, same-file only, matches recorded
via 'fallback' for audit. API failures degrade to non-matches through
onError; both arms share the matcher so the A/B delta stays unbiased.
On by default in live-ab.ts; --no-match-arbiter restores deterministic-only
matching. This targets the 54-69% 'unmatched posted' readings, implausibly
high as true noise.

* [jwbron/eval-measurement-tool] review: split live-ab report shapes/renderers into live-ab-report.ts

CI's max-lines (1000) caught live-ab.ts at 1011 after the repeats and
arbiter work; this worktree cannot run eslint locally (dot-dir ignore).
The report types (ArmRunReport, AbReport, MultiAbReport, gate types) and
both markdown renderers move to a leaf module with no runner dependency;
live-ab.ts re-exports them so the import surface is unchanged.

* [jwbron/eval-measurement-tool] review: checkpoint the repeats artifact after every repeat

A multi-repeat run carries tens of dollars of spend but wrote its artifact
only at the end, so a crash or cancellation on repeat n forfeited repeats
1..n-1 (the exact dies-with-nothing-emitted failure mode the plan forbids;
the workflow's always() artifact upload had nothing to grab). Each completed
repeat now overwrites the out path with the accumulated partial payload,
which aggregate.ts already pools via its repeats-field handling.

* [jwbron/eval-measurement-tool] review: publish the measured noise floor in the report footer

Bought per the memo's item 6: run 29069228968 (--force-arms --repeats 3,
full 14-case corpus, $58.71 under the $60 clamp) gives 6 arm-samples of
one review.md. Measured bands: must-catch recall 54-86%, verdict agreement
75-100%, noise 50-60%, judge quality 0.82-0.86. Every single-run report now
renders these as data with provenance, replacing prose guesswork; a delta
whose arms both sit inside a band is wobble, and the footer points at
--repeats for smaller effects. Pre-arbiter measurement; the weekly drift
run re-measures with the arbiter active.

* [jwbron/eval-measurement-tool] review: raise the drift run's default budget to $85

The noise-floor dispatch showed the live corpus is now 14 cases (the #251
golden re-review set), and the $60 default trimmed case-runs on both arms
(38 and 36 of 42; $58.71 spent at the clamp). $85 covers 6 full arm-runs
at the measured ~$10/arm-run with landing headroom, so the weekly aggregate
never carries budget-skip asymmetry.

* [jwbron/eval-measurement-tool] review: cover the arbiter, checkpoints, and noise floor in the changeset

* [jwbron/eval-measurement-tool] review: eval operator README; drift report opens a visibility PR

workflows/review/eval/README.md is the guide an agent or engineer needs to
run the eval system cold: the three tiers, every live-ab flag and CI entry
point, powered-run recipes with real costs, corpus-growth rules (target the
20-80% band), report-reading guidance anchored on the measured noise floor,
model pins, and the harness's historical limits (nothing before the
2026-07-08 structured-agent architecture runs under it).

The weekly drift run now also opens a PR committing the report markdown and
compact aggregate under .github/review-eval/drift/ (changeset-gate-excluded,
outside the A/B path filter): a job summary nobody opens is not a drift
watch, and merged report PRs accumulate an in-repo time series.

* [jwbron/eval-measurement-tool] review: link the eval operator guide from the main README

* [jwbron/eval-measurement-tool] review: ruler provenance, honest noise-floor statistics, rolling repeat budgets

Findings from an adversarial pass over the instrument's own claims:

Reports now stamp their ruler (matcher configuration + a content hash of
the loaded corpus); the aggregate prints it, warns when a pool mixes
rulers, and the drift series stays interpretable across instrument
upgrades like the arbiter default or corpus growth, which move every rate
without the reviewer changing.

Noise-floor bands gain SD (min/max are extreme-value statistics that only
widen as samples accumulate; mean +/- sd is the stable band) and a loud
case-asymmetry warning: the 2026-07-10 measurement had budget skips, so
its published v1 bands fold case-mix variance in on top of wobble; the
footer constants and provenance string now say so.

Repeat budgets roll: each arm-run's slice is the remaining budget over the
remaining arm-runs, so cheap early arms donate headroom forward instead of
stranding it (fixed slices caused the 38/36-of-42 skip asymmetry that
contaminated the noise-floor run). README gains a statistical-honesty
section: clustered-interval optimism, the repeats-gate relaxation, and the
arbiter's uncalibrated refuse bias.

* [jwbron/eval-measurement-tool] review: checkpoint the repeat reports array, not the scalar count

The per-repeat checkpoint wrote {repeats: <n>} where the recovery path
in aggregate.ts expects {repeats: [reports]}; a crash mid-run therefore
forfeited every completed repeat, which is exactly what the checkpoint
exists to prevent. Write the reports array, matching the final artifact
shape, and lock the checkpoint-to-extractSamples contract with a test.

* [jwbron/eval-measurement-tool] review: merge same-module type imports; pin github-script to v8

Two review nitpicks from the stack: live-match.ts imported two types from
./corpus/loader in separate statements where the rest of the eval dir
merges them, and the sticky-comment step pinned github-script v7 while
node-ci.yml and release.yml pin v8 (ed597411). Align both.

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals

The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING
(full | scoped | flip-gated | fast, default full) dials how much of the
roster a repeat review runs. lib/rereview-mode.ts owns the deterministic
half: content-hashed hunk signatures over the generated-stripped diff,
the hidden review-body fingerprint stamp (survives cache eviction,
dismiss-stale-approvals, and COMMENTED-only histories), the divergence
tripwire that re-arms full review at unreviewed share >= 0.4, and the
plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval,
sparse-PR-then-payload) ship as data in eval/lifecycle/ with a
deterministic replay harness; counters gain costByRereviewDepth so the
live A/B prices scoped against full.

The review.md orchestrator wiring lands in a follow-up commit, on top of
fold-in batch two's re-review accountability surface.

* [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer

The re-review mode dial's flip gate needs 'did the reconciler keep any
blocking thread' as a number it can read from rereview.json, not a label
judgment re-made at verdict time. Additive: the section rendering is
unchanged.

* [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator

Step 1 stages the bot's prior reviews (every state, dismissed and
comment-only included) for the stamp reader. Step 3 runs the
rereview-mode plan CLI after the provenance CLI and maps each depth to
its dispatch and staging: scoped overwrites full-stripped.diff with
scoped.diff so the full roster reads only unseen hunks; flip-gated
dispatches reconcile plus correctness over the scoped diff; fast is
reconcile-only. Step 4 gains the flip rule: a reduced-depth run may
flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0,
and in flip-gated depth a validated blocking finding posts and blocks,
vetoing the flip. Step 6 appends the hidden fingerprint stamp as the
last line of every submitted review body; Steps 7 and 8 skip on
reduced depths; Step 9 notes the stamp, not the cache, is the
tripwire's authoritative fingerprint. The executed plan is copied into
out/ so the run artifact records the depth the counters price.

* [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree

The dispatch-tax trim brought workflows/review into eslint+prettier
scope; format the mode-dial files and the two pre-existing violations
in disciplines.test.ts and finding-schema.ts so CI lints clean.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/rereview-live-corpus] review: re-review coverage for the eval corpus (open-PR cases)

A live-enabled corpus case may now carry a live.rereview block: an
open-PR snapshot with the prior review's unresolved threads (author
replies included), a stamped prior review derived from priorDiff, and
the depth plan the mode dial would compute. The live producer
dispatches the thread-reconciler on such cases at every depth and
sizes the finder roster by the plan; rereview-match.ts scores
thread-resolution accuracy, the flip-gate input, and duplicate
comments on kept threads; live-ab prices a mode via --re-review-mode
on the candidate arm. The deterministic replay learns the same rules:
computeVerdict gains keptBlockingCount (the flip rule as code) and
re-review cases render the accountability section into the planned
body. Ships golden-retention-lifecycle-1/2/3, a sanitized port of the
measured seeded live trial: planted bugs, a bad partial fix with added
bugs, then the full fix that must flip to APPROVE.

* [jwbron/rereview-live-corpus] review: the re-review mode sweep and the under-tripwire pricing case

eval/rereview-sweep.ts runs the working tree's reviewer over the
open-PR corpus cases at every dial setting and prices the dial in one
table: recall, thread resolution, flip-gate correctness, duplicates,
and dollars per mode, with the executed depth per case since the
tripwire may override the dial. golden-retention-fix-push is the
under-threshold case the cheap paths need: a one-hunk fix push
(unreviewed share 0.2) that resolves the prior blocking thread but
plants a fresh defect inside the new hunk, so scoped and flip-gated
must catch it and fast definitionally cannot. No case-format change:
the mode is a run parameter.

* [jwbron/rereview-live-corpus] review: dispatchable CI home for the re-review mode sweep

The per-mode pricing table needs somewhere visible to land: a
workflow_dispatch job (never PR-triggered; sweeps spend model dollars)
that runs rereview-sweep.ts against the chosen branch, tees the
markdown table into the job summary, and uploads the JSON report as an
artifact.

* [jwbron/rereview-live-corpus] review: manual trigger surface for every model-spending eval, and the no-delta short-circuit

From the eval-tuning memo. Labels: rereview-sweep runs the dial sweep
on a PR (sticky comment + summary + artifact), live-judge runs the
judged corpus pass on a PR head, and full-eval now fires on the
labeling event itself instead of waiting for the next push; every
workflow keeps workflow_dispatch for off-PR runs. The A/B gains the
memo's pre-flight identity short-circuit: byte-identical review.md in
both arms posts a no-reviewable-delta verdict and runs nothing, with
--force-arms preserved for deliberate wobble controls.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* [jwbron/eval-measurement-tool] review: repeat aggregation, --repeats powered runs, drift watch, sql-index case fix

The eval instrument's percentage deltas sit below the one-case noise floor;
this lands the memo's measurement items. eval/aggregate.ts pools N report
artifacts into per-case pass rates with Wilson intervals and pooled rows
(reproduces the 07-10 cumulative numbers exactly); live-ab.ts --repeats runs
one powered dispatch with a strict-majority adversarial gate; the weekly
review-eval-drift.yml workflow runs full corpus x3 on main and publishes the
aggregate; identical-arm pools render noise-floor bands as data.

incident-sql-missing-index diagnosed against all 28 recorded arm-runs: the
reviewer found the missing index every single time; the 8/16 catch rate was
the spec accepting only the migration-file anchor while the reviewer anchored
the same blocking finding at the hot query (11 runs) or had it provenance-
dropped on a mis-anchor (2 runs). Specs gain altLocations; the case accepts
the query-site anchor and its residual misses now classify found-but-dropped
(the anchor-snap defect class), not recall.

* [jwbron/eval-measurement-tool] review: implement the fallback match arbiter on the pinned Haiku snapshot

Fills the MatchFallback seam in live-match.ts: when a spec stays unmatched
after the deterministic pass and a posted candidate shares its file, a
pinned claude-haiku-4-5-20251001 answers one yes/no question (same defect,
same root cause?), biased to no since a false yes inflates recall. All seam
guard rails inherited: capped per case, same-file only, matches recorded
via 'fallback' for audit. API failures degrade to non-matches through
onError; both arms share the matcher so the A/B delta stays unbiased.
On by default in live-ab.ts; --no-match-arbiter restores deterministic-only
matching. This targets the 54-69% 'unmatched posted' readings, implausibly
high as true noise.

* [jwbron/eval-measurement-tool] review: split live-ab report shapes/renderers into live-ab-report.ts

CI's max-lines (1000) caught live-ab.ts at 1011 after the repeats and
arbiter work; this worktree cannot run eslint locally (dot-dir ignore).
The report types (ArmRunReport, AbReport, MultiAbReport, gate types) and
both markdown renderers move to a leaf module with no runner dependency;
live-ab.ts re-exports them so the import surface is unchanged.

* [jwbron/eval-measurement-tool] review: checkpoint the repeats artifact after every repeat

A multi-repeat run carries tens of dollars of spend but wrote its artifact
only at the end, so a crash or cancellation on repeat n forfeited repeats
1..n-1 (the exact dies-with-nothing-emitted failure mode the plan forbids;
the workflow's always() artifact upload had nothing to grab). Each completed
repeat now overwrites the out path with the accumulated partial payload,
which aggregate.ts already pools via its repeats-field handling.

* [jwbron/eval-measurement-tool] review: publish the measured noise floor in the report footer

Bought per the memo's item 6: run 29069228968 (--force-arms --repeats 3,
full 14-case corpus, $58.71 under the $60 clamp) gives 6 arm-samples of
one review.md. Measured bands: must-catch recall 54-86%, verdict agreement
75-100%, noise 50-60%, judge quality 0.82-0.86. Every single-run report now
renders these as data with provenance, replacing prose guesswork; a delta
whose arms both sit inside a band is wobble, and the footer points at
--repeats for smaller effects. Pre-arbiter measurement; the weekly drift
run re-measures with the arbiter active.

* [jwbron/eval-measurement-tool] review: raise the drift run's default budget to $85

The noise-floor dispatch showed the live corpus is now 14 cases (the #251
golden re-review set), and the $60 default trimmed case-runs on both arms
(38 and 36 of 42; $58.71 spent at the clamp). $85 covers 6 full arm-runs
at the measured ~$10/arm-run with landing headroom, so the weekly aggregate
never carries budget-skip asymmetry.

* [jwbron/eval-measurement-tool] review: cover the arbiter, checkpoints, and noise floor in the changeset

* [jwbron/eval-measurement-tool] review: eval operator README; drift report opens a visibility PR

workflows/review/eval/README.md is the guide an agent or engineer needs to
run the eval system cold: the three tiers, every live-ab flag and CI entry
point, powered-run recipes with real costs, corpus-growth rules (target the
20-80% band), report-reading guidance anchored on the measured noise floor,
model pins, and the harness's historical limits (nothing before the
2026-07-08 structured-agent architecture runs under it).

The weekly drift run now also opens a PR committing the report markdown and
compact aggregate under .github/review-eval/drift/ (changeset-gate-excluded,
outside the A/B path filter): a job summary nobody opens is not a drift
watch, and merged report PRs accumulate an in-repo time series.

* [jwbron/eval-measurement-tool] review: link the eval operator guide from the main README

* [jwbron/eval-measurement-tool] review: ruler provenance, honest noise-floor statistics, rolling repeat budgets

Findings from an adversarial pass over the instrument's own claims:

Reports now stamp their ruler (matcher configuration + a content hash of
the loaded corpus); the aggregate prints it, warns when a pool mixes
rulers, and the drift series stays interpretable across instrument
upgrades like the arbiter default or corpus growth, which move every rate
without the reviewer changing.

Noise-floor bands gain SD (min/max are extreme-value statistics that only
widen as samples accumulate; mean +/- sd is the stable band) and a loud
case-asymmetry warning: the 2026-07-10 measurement had budget skips, so
its published v1 bands fold case-mix variance in on top of wobble; the
footer constants and provenance string now say so.

Repeat budgets roll: each arm-run's slice is the remaining budget over the
remaining arm-runs, so cheap early arms donate headroom forward instead of
stranding it (fixed slices caused the 38/36-of-42 skip asymmetry that
contaminated the noise-floor run). README gains a statistical-honesty
section: clustered-interval optimism, the repeats-gate relaxation, and the
arbiter's uncalibrated refuse bias.

* [jwbron/anchor-snap-provenance] review: anchor-snap fallback in the change-provenance gate

The reviewer produces right-file, right-mechanism findings at wrong line
numbers (it appears to sometimes count unified-diff text lines instead of
file lines: observed anchors at line 24 of an 18-line file and line 8 of
a 3-line file), and the provenance gate then drops them; since only
surviving blocking labels feed the computed verdict, a correct blocking
finding dying at the gate can flip REQUEST_CHANGES to APPROVE. Observed
on adversarial-injection-approve, golden-request-changes-authz, the
sql-index replay, and production main's own baseline arm.

Before setting a line-anchored finding aside, the gate now snaps it to
the nearest changed line in the same file under two windows: a 3-line
near-miss window (the unified diff's context width) and an overflow
window for anchors past every shown line by no more than the file's
diff-text overhead (exactly the counting mis-anchor's overshoot bound).
Ties break toward the lower line. Snapped findings keep their severity,
post at the snapped anchor, and are recorded for audit (out/snapped.json
in production, snappedByProvenance in eval results); findings outside
both windows keep today's set-aside behavior. provenance.json carries a
precomputed per-file snap lookup so review.md's gate stays a dictionary
lookup. The live A/B emulates each arm's own review.md gate version,
keyed on the anchor-snap marker, so the baseline arm replays the
pre-snap gate and the powered run prices this change.

* [jwbron/anchor-snap-provenance] review: pin anchor-snap in a deterministic smoke case

provenance-anchor-snap-rescued replays the observed production anatomy
in the per-push CI gate: a correct blocking finding anchored at the
diff-text line (past the end of a short file, at the exact overflow
edge) snaps to the nearest changed line, posts there, and drives
REQUEST_CHANGES, while a far-anchored pre-existing observation in the
same run is still set aside unposted. Deterministic and smoke-tagged
only, so the live ruler (corpus stamp over live-tagged cases) is
untouched and the in-flight powered run stays comparable. Also
documents how to audit snap records (from/to distance separates the
near-miss and overflow classes).

* [jwbron/anchor-snap-provenance] review: overflow-snap only past the file's real end; drop LEFT-side snapping

The overflow window could rewrite a genuine finding about unshown code
below a short hunk into a posted blocking comment (a small hunk in a
longer file leaves real lines between lastShownLine and the overhead
bound). The window now additionally requires the anchor to be past the
file's actual end (a line that does not exist): the provenance CLI reads
each changed file's real length from the checkout, live eval cases
hydrate lengths from their tree, and deterministic cases declare them.
A file whose length is unknown gets no overflow window (near-miss only).
LEFT-side anchors no longer snap anywhere, matching the RIGHT-side-only
snap table the production gate reads, so the eval emulation never prices
a gate production does not have. Serialized snap tables are now asserted
in the CLI tests.

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals

The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING
(full | scoped | flip-gated | fast, default full) dials how much of the
roster a repeat review runs. lib/rereview-mode.ts owns the deterministic
half: content-hashed hunk signatures over the generated-stripped diff,
the hidden review-body fingerprint stamp (survives cache eviction,
dismiss-stale-approvals, and COMMENTED-only histories), the divergence
tripwire that re-arms full review at unreviewed share >= 0.4, and the
plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval,
sparse-PR-then-payload) ship as data in eval/lifecycle/ with a
deterministic replay harness; counters gain costByRereviewDepth so the
live A/B prices scoped against full.

The review.md orchestrator wiring lands in a follow-up commit, on top of
fold-in batch two's re-review accountability surface.

* [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer

The re-review mode dial's flip gate needs 'did the reconciler keep any
blocking thread' as a number it can read from rereview.json, not a label
judgment re-made at verdict time. Additive: the section rendering is
unchanged.

* [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator

Step 1 stages the bot's prior reviews (every state, dismissed and
comment-only included) for the stamp reader. Step 3 runs the
rereview-mode plan CLI after the provenance CLI and maps each depth to
its dispatch and staging: scoped overwrites full-stripped.diff with
scoped.diff so the full roster reads only unseen hunks; flip-gated
dispatches reconcile plus correctness over the scoped diff; fast is
reconcile-only. Step 4 gains the flip rule: a reduced-depth run may
flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0,
and in flip-gated depth a validated blocking finding posts and blocks,
vetoing the flip. Step 6 appends the hidden fingerprint stamp as the
last line of every submitted review body; Steps 7 and 8 skip on
reduced depths; Step 9 notes the stamp, not the cache, is the
tripwire's authoritative fingerprint. The executed plan is copied into
out/ so the run artifact records the depth the counters price.

* [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree

The dispatch-tax trim brought workflows/review into eslint+prettier
scope; format the mode-dial files and the two pre-existing violations
in disciplines.test.ts and finding-schema.ts so CI lints clean.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/rereview-live-corpus] review: re-review coverage for the eval corpus (open-PR cases)

A live-enabled corpus case may now carry a live.rereview block: an
open-PR snapshot with the prior review's unresolved threads (author
replies included), a stamped prior review derived from priorDiff, and
the depth plan the mode dial would compute. The live producer
dispatches the thread-reconciler on such cases at every depth and
sizes the finder roster by the plan; rereview-match.ts scores
thread-resolution accuracy, the flip-gate input, and duplicate
comments on kept threads; live-ab prices a mode via --re-review-mode
on the candidate arm. The deterministic replay learns the same rules:
computeVerdict gains keptBlockingCount (the flip rule as code) and
re-review cases render the accountability section into the planned
body. Ships golden-retention-lifecycle-1/2/3, a sanitized port of the
measured seeded live trial: planted bugs, a bad partial fix with added
bugs, then the full fix that must flip to APPROVE.

* [jwbron/rereview-live-corpus] review: the re-review mode sweep and the under-tripwire pricing case

eval/rereview-sweep.ts runs the working tree's reviewer over the
open-PR corpus cases at every dial setting and prices the dial in one
table: recall, thread resolution, flip-gate correctness, duplicates,
and dollars per mode, with the executed depth per case since the
tripwire may override the dial. golden-retention-fix-push is the
under-threshold case the cheap paths need: a one-hunk fix push
(unreviewed share 0.2) that resolves the prior blocking thread but
plants a fresh defect inside the new hunk, so scoped and flip-gated
must catch it and fast definitionally cannot. No case-format change:
the mode is a run parameter.

* [jwbron/rereview-live-corpus] review: dispatchable CI home for the re-review mode sweep

The per-mode pricing table needs somewhere visible to land: a
workflow_dispatch job (never PR-triggered; sweeps spend model dollars)
that runs rereview-sweep.ts against the chosen branch, tees the
markdown table into the job summary, and uploads the JSON report as an
artifact.

* [jwbron/rereview-live-corpus] review: manual trigger surface for every model-spending eval, and the no-delta short-circuit

From the eval-tuning memo. Labels: rereview-sweep runs the dial sweep
on a PR (sticky comment + summary + artifact), live-judge runs the
judged corpus pass on a PR head, and full-eval now fires on the
labeling event itself instead of waiting for the next push; every
workflow keeps workflow_dispatch for off-PR runs. The A/B gains the
memo's pre-flight identity short-circuit: byte-identical review.md in
both arms posts a no-reviewable-delta verdict and runs nothing, with
--force-arms preserved for deliberate wobble controls.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* [jwbron/eval-measurement-tool] review: repeat aggregation, --repeats powered runs, drift watch, sql-index case fix

The eval instrument's percentage deltas sit below the one-case noise floor;
this lands the memo's measurement items. eval/aggregate.ts pools N report
artifacts into per-case pass rates with Wilson intervals and pooled rows
(reproduces the 07-10 cumulative numbers exactly); live-ab.ts --repeats runs
one powered dispatch with a strict-majority adversarial gate; the weekly
review-eval-drift.yml workflow runs full corpus x3 on main and publishes the
aggregate; identical-arm pools render noise-floor bands as data.

incident-sql-missing-index diagnosed against all 28 recorded arm-runs: the
reviewer found the missing index every single time; the 8/16 catch rate was
the spec accepting only the migration-file anchor while the reviewer anchored
the same blocking finding at the hot query (11 runs) or had it provenance-
dropped on a mis-anchor (2 runs). Specs gain altLocations; the case accepts
the query-site anchor and its residual misses now classify found-but-dropped
(the anchor-snap defect class), not recall.

* [jwbron/eval-measurement-tool] review: implement the fallback match arbiter on the pinned Haiku snapshot

Fills the MatchFallback seam in live-match.ts: when a spec stays unmatched
after the deterministic pass and a posted candidate shares its file, a
pinned claude-haiku-4-5-20251001 answers one yes/no question (same defect,
same root cause?), biased to no since a false yes inflates recall. All seam
guard rails inherited: capped per case, same-file only, matches recorded
via 'fallback' for audit. API failures degrade to non-matches through
onError; both arms share the matcher so the A/B delta stays unbiased.
On by default in live-ab.ts; --no-match-arbiter restores deterministic-only
matching. This targets the 54-69% 'unmatched posted' readings, implausibly
high as true noise.

* [jwbron/eval-measurement-tool] review: split live-ab report shapes/renderers into live-ab-report.ts

CI's max-lines (1000) caught live-ab.ts at 1011 after the repeats and
arbiter work; this worktree cannot run eslint locally (dot-dir ignore).
The report types (ArmRunReport, AbReport, MultiAbReport, gate types) and
both markdown renderers move to a leaf module with no runner dependency;
live-ab.ts re-exports them so the import surface is unchanged.

* [jwbron/eval-measurement-tool] review: checkpoint the repeats artifact after every repeat

A multi-repeat run carries tens of dollars of spend but wrote its artifact
only at the end, so a crash or cancellation on repeat n forfeited repeats
1..n-1 (the exact dies-with-nothing-emitted failure mode the plan forbids;
the workflow's always() artifact upload had nothing to grab). Each completed
repeat now overwrites the out path with the accumulated partial payload,
which aggregate.ts already pools via its repeats-field handling.

* [jwbron/eval-measurement-tool] review: publish the measured noise floor in the report footer

Bought per the memo's item 6: run 29069228968 (--force-arms --repeats 3,
full 14-case corpus, $58.71 under the $60 clamp) gives 6 arm-samples of
one review.md. Measured bands: must-catch recall 54-86%, verdict agreement
75-100%, noise 50-60%, judge quality 0.82-0.86. Every single-run report now
renders these as data with provenance, replacing prose guesswork; a delta
whose arms both sit inside a band is wobble, and the footer points at
--repeats for smaller effects. Pre-arbiter measurement; the weekly drift
run re-measures with the arbiter active.

* [jwbron/eval-measurement-tool] review: raise the drift run's default budget to $85

The noise-floor dispatch showed the live corpus is now 14 cases (the #251
golden re-review set), and the $60 default trimmed case-runs on both arms
(38 and 36 of 42; $58.71 spent at the clamp). $85 covers 6 full arm-runs
at the measured ~$10/arm-run with landing headroom, so the weekly aggregate
never carries budget-skip asymmetry.

* [jwbron/eval-measurement-tool] review: cover the arbiter, checkpoints, and noise floor in the changeset

* [jwbron/eval-measurement-tool] review: eval operator README; drift report opens a visibility PR

workflows/review/eval/README.md is the guide an agent or engineer needs to
run the eval system cold: the three tiers, every live-ab flag and CI entry
point, powered-run recipes with real costs, corpus-growth rules (target the
20-80% band), report-reading guidance anchored on the measured noise floor,
model pins, and the harness's historical limits (nothing before the
2026-07-08 structured-agent architecture runs under it).

The weekly drift run now also opens a PR committing the report markdown and
compact aggregate under .github/review-eval/drift/ (changeset-gate-excluded,
outside the A/B path filter): a job summary nobody opens is not a drift
watch, and merged report PRs accumulate an in-repo time series.

* [jwbron/eval-measurement-tool] review: link the eval operator guide from the main README

* [jwbron/eval-measurement-tool] review: ruler provenance, honest noise-floor statistics, rolling repeat budgets

Findings from an adversarial pass over the instrument's own claims:

Reports now stamp their ruler (matcher configuration + a content hash of
the loaded corpus); the aggregate prints it, warns when a pool mixes
rulers, and the drift series stays interpretable across instrument
upgrades like the arbiter default or corpus growth, which move every rate
without the reviewer changing.

Noise-floor bands gain SD (min/max are extreme-value statistics that only
widen as samples accumulate; mean +/- sd is the stable band) and a loud
case-asymmetry warning: the 2026-07-10 measurement had budget skips, so
its published v1 bands fold case-mix variance in on top of wobble; the
footer constants and provenance string now say so.

Repeat budgets roll: each arm-run's slice is the remaining budget over the
remaining arm-runs, so cheap early arms donate headroom forward instead of
stranding it (fixed slices caused the 38/36-of-42 skip asymmetry that
contaminated the noise-floor run). README gains a statistical-honesty
section: clustered-interval optimism, the repeats-gate relaxation, and the
arbiter's uncalibrated refuse bias.

* [jwbron/anchor-snap-provenance] review: anchor-snap fallback in the change-provenance gate

The reviewer produces right-file, right-mechanism findings at wrong line
numbers (it appears to sometimes count unified-diff text lines instead of
file lines: observed anchors at line 24 of an 18-line file and line 8 of
a 3-line file), and the provenance gate then drops them; since only
surviving blocking labels feed the computed verdict, a correct blocking
finding dying at the gate can flip REQUEST_CHANGES to APPROVE. Observed
on adversarial-injection-approve, golden-request-changes-authz, the
sql-index replay, and production main's own baseline arm.

Before setting a line-anchored finding aside, the gate now snaps it to
the nearest changed line in the same file under two windows: a 3-line
near-miss window (the unified diff's context width) and an overflow
window for anchors past every shown line by no more than the file's
diff-text overhead (exactly the counting mis-anchor's overshoot bound).
Ties break toward the lower line. Snapped findings keep their severity,
post at the snapped anchor, and are recorded for audit (out/snapped.json
in production, snappedByProvenance in eval results); findings outside
both windows keep today's set-aside behavior. provenance.json carries a
precomputed per-file snap lookup so review.md's gate stays a dictionary
lookup. The live A/B emulates each arm's own review.md gate version,
keyed on the anchor-snap marker, so the baseline arm replays the
pre-snap gate and the powered run prices this change.

* [jwbron/anchor-snap-provenance] review: pin anchor-snap in a deterministic smoke case

provenance-anchor-snap-rescued replays the observed production anatomy
in the per-push CI gate: a correct blocking finding anchored at the
diff-text line (past the end of a short file, at the exact overflow
edge) snaps to the nearest changed line, posts there, and drives
REQUEST_CHANGES, while a far-anchored pre-existing observation in the
same run is still set aside unposted. Deterministic and smoke-tagged
only, so the live ruler (corpus stamp over live-tagged cases) is
untouched and the in-flight powered run stays comparable. Also
documents how to audit snap records (from/to distance separates the
near-miss and overflow classes).

* [jwbron/diff-line-annotation] review: line-number-annotated staged diffs, anchors read not counted

The mis-anchor pathology anchor-snap repairs downstream exists because
the staged diff makes the model count lines. annotateDiffLineNumbers
prefixes every hunk content line with its real line number (+/context
lines carry the NEW-file number, - lines the OLD-file number) while
keeping the diff marker in column one, so annotated text still splits
into sections. The provenance CLI writes full-stripped-annotated.diff
beside the raw stripped diff and gains an 'annotate <in> <out>'
subcommand for pr-annotated.diff (Phase 1) and the scoped-depth
refreshes. Every finding-producing reviewer reads the annotated copy
and takes anchor.line from the printed number; everything that parses
a diff (provenance, re-review fingerprints, scoped staging,
pattern-triage, claim-validator) keeps reading the raw files, so hunk
signatures never shift. Eval staging writes the annotated siblings for
both arms unconditionally; only a review.md that names them reads
them, so the A/B prices this as a pure prompt delta. Reports gain the
anchor-fidelity observable: per-case snap counts and a pooled
'Findings anchor-snapped' row (version-tolerant of older artifacts);
if annotation works, candidate-arm snaps fall to zero with the
anchor-snap gate remaining as backstop.

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/rereview-mode-dial] review: the re-review mode dial: ROUTING parsing, deterministic depth lib, lifecycle evals

The runs-per-PR cost lever. A per-repo 're-review' line in ROUTING
(full | scoped | flip-gated | fast, default full) dials how much of the
roster a repeat review runs. lib/rereview-mode.ts owns the deterministic
half: content-hashed hunk signatures over the generated-stripped diff,
the hidden review-body fingerprint stamp (survives cache eviction,
dismiss-stale-approvals, and COMMENTED-only histories), the divergence
tripwire that re-arms full review at unreviewed share >= 0.4, and the
plan/stamp CLI. Adversarial lifecycle cases (rewrite-after-approval,
sparse-PR-then-payload) ship as data in eval/lifecycle/ with a
deterministic replay harness; counters gain costByRereviewDepth so the
live A/B prices scoped against full.

The review.md orchestrator wiring lands in a follow-up commit, on top of
fold-in batch two's re-review accountability surface.

* [jwbron/rereview-mode-dial] review: expose keptBlockingCount from the accountability renderer

The re-review mode dial's flip gate needs 'did the reconciler keep any
blocking thread' as a number it can read from rereview.json, not a label
judgment re-made at verdict time. Additive: the section rendering is
unchanged.

* [jwbron/rereview-mode-dial] review: wire the re-review mode dial into the orchestrator

Step 1 stages the bot's prior reviews (every state, dismissed and
comment-only included) for the stamp reader. Step 3 runs the
rereview-mode plan CLI after the provenance CLI and maps each depth to
its dispatch and staging: scoped overwrites full-stripped.diff with
scoped.diff so the full roster reads only unseen hunks; flip-gated
dispatches reconcile plus correctness over the scoped diff; fast is
reconcile-only. Step 4 gains the flip rule: a reduced-depth run may
flip a stamped REQUEST_CHANGES to APPROVE only at keptBlockingCount 0,
and in flip-gated depth a validated blocking finding posts and blocks,
vetoing the flip. Step 6 appends the hidden fingerprint stamp as the
last line of every submitted review body; Steps 7 and 8 skip on
reduced depths; Step 9 notes the stamp, not the cache, is the
tripwire's authoritative fingerprint. The executed plan is copied into
out/ so the run artifact records the depth the counters price.

* [jwbron/rereview-mode-dial] review: prettier-format the mode-dial files for the newly-linted workflows tree

The dispatch-tax trim brought workflows/review into eslint+prettier
scope; format the mode-dial files and the two pre-existing violations
in disciplines.test.ts and finding-schema.ts so CI lints clean.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwbron/rereview-live-corpus] review: re-review coverage for the eval corpus (open-PR cases)

A live-enabled corpus case may now carry a live.rereview block: an
open-PR snapshot with the prior review's unresolved threads (author
replies included), a stamped prior review derived from priorDiff, and
the depth plan the mode dial would compute. The live producer
dispatches the thread-reconciler on such cases at every depth and
sizes the finder roster by the plan; rereview-match.ts scores
thread-resolution accuracy, the flip-gate input, and duplicate
comments on kept threads; live-ab prices a mode via --re-review-mode
on the candidate arm. The deterministic replay learns the same rules:
computeVerdict gains keptBlockingCount (the flip rule as code) and
re-review cases render the accountability section into the planned
body. Ships golden-retention-lifecycle-1/2/3, a sanitized port of the
measured seeded live trial: planted bugs, a bad partial fix with added
bugs, then the full fix that must flip to APPROVE.

* [jwbron/rereview-live-corpus] review: the re-review mode sweep and the under-tripwire pricing case

eval/rereview-sweep.ts runs the working tree's reviewer over the
open-PR corpus cases at every dial setting and prices the dial in one
table: recall, thread resolution, flip-gate correctness, duplicates,
and dollars per mode, with the executed depth per case since the
tripwire may override the dial. golden-retention-fix-push is the
under-threshold case the cheap paths need: a one-hunk fix push
(unreviewed share 0.2) that resolves the prior blocking thread but
plants a fresh defect inside the new hunk, so scoped and flip-gated
must catch it and fast definitionally cannot. No case-format change:
the mode is a run parameter.

* [jwbron/rereview-live-corpus] review: dispatchable CI home for the re-review mode sweep

The per-mode pricing table needs somewhere visible to land: a
workflow_dispatch job (never PR-triggered; sweeps spend model dollars)
that runs rereview-sweep.ts against the chosen branch, tees the
markdown table into the job summary, and uploads the JSON report as an
artifact.

* [jwbron/rereview-live-corpus] review: manual trigger surface for every model-spending eval, and the no-delta short-circuit

From the eval-tuning memo. Labels: rereview-sweep runs the dial sweep
on a PR (sticky comment + summary + artifact), live-judge runs the
judged corpus pass on a PR head, and full-eval now fires on the
labeling event itself instead of waiting for the next push; every
workflow keeps workflow_dispatch for off-PR runs. The A/B gains the
memo's pre-flight identity short-circuit: byte-identical review.md in
both arms posts a no-reviewable-delta verdict and runs nothing, with
--force-arms preserved for deliberate wobble controls.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* [jwbron/eval-measurement-tool] review: repeat aggregation, --repeats powered runs, drift watch, sql-index case fix

The eval instrument's percentage deltas sit below the one-case noise floor;
this lands the memo's measurement items. eval/aggregate.ts pools N report
artifacts into per-case pass rates with Wilson intervals and pooled rows
(reproduces the 07-10 cumulative numbers exactly); live-ab.ts --repeats runs
one powered dispatch with a strict-majority adversarial gate; the weekly
review-eval-drift.yml workflow runs full corpus x3 on main and publishes the
aggregate; identical-arm pools render noise-floor bands as data.

incident-sql-missing-index diagnosed against all 28 recorded arm-runs: the
reviewer found the missing index every single time; the 8/16 catch rate was
the spec accepting only the migration-file anchor while the reviewer anchored
the same blocking finding at the hot query (11 runs) or had it provenance-
dropped on a mis-anchor (2 runs). Specs gain altLocations; the case accepts
the query-site anchor and its residual misses now classify found-but-dropped
(the anchor-snap defect class), not recall.

* [jwbron/eval-measurement-tool] review: implement the fallback match arbiter on the pinned Haiku snapshot

Fills the MatchFallback seam in live-match.ts: when a spec stays unmatched
after the deterministic pass and a posted candidate shares its file, a
pinned claude-haiku-4-5-20251001 answers one yes/no question (same defect,
same root cause?), biased to no since a false yes inflates recall. All seam
guard rails inherited: capped per case, same-file only, matches recorded
via 'fallback' for audit. API failures degrade to non-matches through
onError; both arms share the matcher so the A/B delta stays unbiased.
On by default in live-ab.ts; --no-match-arbiter restores deterministic-only
matching. This targets the 54-69% 'unmatched posted' readings, implausibly
high as true noise.

* [jwbron/eval-measurement-tool] review: split live-ab report shapes/renderers into live-ab-report.ts

CI's max-lines (1000) caught live-ab.ts at 1011 after the repeats and
arbiter work; this worktree cannot run eslint locally (dot-dir ignore).
The report types (ArmRunReport, AbReport, MultiAbReport, gate types) and
both markdown renderers move to a leaf module with no runner dependency;
live-ab.ts re-exports them so the import surface is unchanged.

* [jwbron/eval-measurement-tool] review: checkpoint the repeats artifact after every repeat

A multi-repeat run carries tens of dollars of spend but wrote its artifact
only at the end, so a crash or cancellation on repeat n forfeited repeats
1..n-1 (the exact dies-with-nothing-emitted failure mode the plan forbids;
the workflow's always() artifact upload had nothing to grab). Each completed
repeat now overwrites the out path with the accumulated partial payload,
which aggregate.ts already pools via its repeats-field handling.

* [jwbron/eval-measurement-tool] review: publish the measured noise floor in the report footer

Bought per the memo's item 6: run 29069228968 (--force-arms --repeats 3,
full 14-case corpus, $58.71 under the $60 clamp) gives 6 arm-samples of
one review.md. Measured bands: must-catch recall 54-86%, verdict agreement
75-100%, noise 50-60%, judge quality 0.82-0.86. Every single-run report now
renders these as data with provenance, replacing prose guesswork; a delta
whose arms both sit inside a band is wobble, and the footer points at
--repeats for smaller effects. Pre-arbiter measurement; the weekly drift
run re-measures with the arbiter active.

* [jwbron/eval-measurement-tool] review: raise the drift run's default budget to $85

The noise-floor dispatch showed the live corpus is now 14 cases (the #251
golden re-review set), and the $60 default trimmed case-runs on both arms
(38 and 36 of 42; $58.71 spent at the clamp). $85 covers 6 full arm-runs
at the measured ~$10/arm-run with landing headroom, so the weekly aggregate
never carries budget-skip asymmetry.

* [jwbron/eval-measurement-tool] review: cover the arbiter, checkpoints, and noise floor in the changeset

* [jwbron/eval-measurement-tool] review: eval operator README; drift report opens a visibility PR

workflows/review/eval/README.md is the guide an agent or engineer needs to
run the eval system cold: the three tiers, every live-ab flag and CI entry
point, powered-run recipes with real costs, corpus-growth rules (target the
20-80% band), report-reading guidance anchored on the measured noise floor,
model pins, and the harness's historical limits (nothing before the
2026-07-08 structured-agent architecture runs under it).

The weekly drift run now also opens a PR committing the report markdown and
compact aggregate under .github/review-eval/drift/ (changeset-gate-excluded,
outside the A/B path filter): a job summary nobody opens is not a drift
watch, and merged report PRs accumulate an in-repo time series.

* [jwbron/eval-measurement-tool] review: link the eval operator guide from the main README

* [jwbron/eval-measurement-tool] review: ruler provenance, honest noise-floor statistics, rolling repeat budgets

Findings from an adversarial pass over the instrument's own claims:

Reports now stamp their ruler (matcher configuration + a content hash of
the loaded corpus); the aggregate prints it, warns when a pool mixes
rulers, and the drift series stays interpretable across instrument
upgrades like the arbiter default or corpus growth, which move every rate
without the reviewer changing.

Noise-floor bands gain SD (min/max are extreme-value statistics that only
widen as samples accumulate; mean +/- sd is the stable band) and a loud
case-asymmetry warning: the 2026-07-10 measurement had budget skips, so
its published v1 bands fold case-mix variance in on top of wobble; the
footer constants and provenance string now say so.

Repeat budgets roll: each arm-run's slice is the remaining budget over the
remaining arm-runs, so cheap early arms donate headroom forward instead of
stranding it (fixed slices caused the 38/36-of-42 skip asymmetry that
contaminated the noise-floor run). README gains a statistical-honesty
section: clustered-interval optimism, the repeats-gate relaxation, and the
arbiter's uncalibrated refuse bias.

* [jwbron/anchor-snap-provenance] review: anchor-snap fallback in the change-provenance gate

The reviewer produces right-file, right-mechanism findings at wrong line
numbers (it appears to sometimes count unified-diff text lines instead of
file lines: observed anchors at line 24 of an 18-line file and line 8 of
a 3-line file), and the provenance gate then drops them; since only
surviving blocking labels feed the computed verdict, a correct blocking
finding dying at the gate can flip REQUEST_CHANGES to APPROVE. Observed
on adversarial-injection-approve, golden-request-changes-authz, the
sql-index replay, and production main's own baseline arm.

Before setting a line-anchored finding aside, the gate now snaps it to
the nearest changed line in the same file under two windows: a 3-line
near-miss window (the unified diff's context width) and an overflow
window for anchors past every shown line by no more than the file's
diff-text overhead (exactly the counting mis-anchor's overshoot bound).
Ties break toward the lower line. Snapped findings keep their severity,
post at the snapped anchor, and are recorded for audit (out/snapped.json
in production, snappedByProvenance in eval results); findings outside
both windows keep today's set-aside behavior. provenance.json carries a
precomputed per-file snap lookup so review.md's gate stays a dictionary
lookup. The live A/B emulates each arm's own review.md gate version,
keyed on the anchor-snap marker, so the baseline arm replays the
pre-snap gate and the powered run prices this change.

* [jwbron/anchor-snap-provenance] review: pin anchor-snap in a deterministic smoke case

provenance-anchor-snap-rescued replays the observed production anatomy
in the per-push CI gate: a correct blocking finding anchored at the
diff-text line (past the end of a short file, at the exact overflow
edge) snaps to the nearest changed line, posts there, and drives
REQUEST_CHANGES, while a far-anchored pre-existing observation in the
same run is still set aside unposted. Deterministic and smoke-tagged
only, so the live ruler (corpus stamp over live-tagged cases) is
untouched and the in-flight powered run stays comparable. Also
documents how to audit snap records (from/to distance separates the
near-miss and overflow classes).

* [jwbron/eval-case-selection] review: exact --cases selection, fail-fast on unknown ids

The smoke scope filtered the corpus before the case filter, so a
powered dispatch naming a non-smoke case silently dropped it: the
anchor-snap pricing run named two cases and the paid report covered
one without saying so. --cases is now an exact selection (bypasses
--smoke-only, preserves order, dedupes) and any id matching no live
case throws before a single model call. The workflow needs no change;
it already passes both flags and the case list now wins.

… the cap (#249)

* [jwbron/live-eval-corpus] review: live-enabled corpus format and ten live cases

Phase 1 of the live A/B eval plan (#232). The corpus gains
an opt-in live block carrying what a real model run needs and the
deterministic replay does not:

- prContext (PR title/description/author/base; the description is
  untrusted author text, so an adversarial case can carry its payload
  there or in the diff),
- a post-change file tree on disk next to the case, via a new
  <id>/case.json + <id>/tree/ layout that coexists with flat <id>.json
  (a directory containing case.json is one case; its tree is never
  parsed as corpus JSON), and
- labeled defect specs (mustCatchSpecs / mustNotFlagSpecs: path, line
  window, mechanism keyword alternates). Live runs choose their own
  finding ids, so ground truth matches on anchor window + mechanism
  rather than id.

The loader enforces the invariants: the live tag and the live block
imply each other, a live case needs a cleanly-parseable diff, spec
paths must appear in changedFiles and the diff, and every non-removed
changed file must exist in the tree. loadLiveCorpus() returns the
subset. The live half lives in corpus/live.ts; loader.ts re-exports it
as the single public surface.

Ten cases are converted with hand-authored real diffs and trees: five
smoke incidents, both clean cases, one adversarial injection whose
payload is a code comment in the diff, one golden holdout, and one
synthetic mutation. Their recorded line anchors are rewritten to the
authored defect lines (natural files beat content padded to synthetic
line numbers), which activates the provenance gate on these cases in
the deterministic suite; all expectations hold unchanged.

* [jwbron/live-eval-producer-staging] review: live-producer prompt extraction and case staging

Phases 2a/2b of the live A/B eval plan (#232), stacked on the Phase 1
corpus format (#233).

agent-extract.ts turns review.md's '## agent:' sections into data
(name, description, pinned model, prompt body). It takes the markdown
as a string with no fs access, because the baseline arm of an A/B
reads the merge-base review.md via git show. Parsing is strict; a
malformed or model-less section throws listing every problem, since a
silently dropped agent would skew an eval arm without failing it. An
integration test extracts the real review.md (21 agents) and pins the
staging-root reference so a future rename fails a test instead of
silently staging nothing.

live-stage.ts materializes the production staging layout for one
live-enabled corpus case: pr-context.json (review.md Step 1's shape,
synthetic identity fields), full.diff / pr.diff / full-stripped.diff
(all the case diff; corpus diffs carry no generated files and no
pattern-triage pass runs), files.json + review-files.json with the
hasPatch cross-check derived from the diff parse, provenance.json,
routing.json from the deterministic router, an out/ directory, and
the post-change checkout copied from the case tree.
rewriteAgentPrompt swaps the production staging root for the staged
context dir. Everything sits behind an injected-fs seam and is
memfs-tested.

Model dispatch (phase 2c) is deliberately absent; it needs the Agent
SDK dependency decision and arrives separately.

* [jwbron/live-eval-producer-staging] review: the live producer and SDK runner (phase 2c)

live-producer.ts runs the live roster over one staged case behind an
injected LiveAgentRunner seam (the judge.ts pattern), so its logic is
stub-tested with zero model calls. Roster: the default finders
(correctness-reviewer, skill-auditor) plus the router's lensesToSpawn;
no pattern-triage or thread-reconciler in eval. It maps all three
sub-agent output contracts into the shapes the deterministic runner
consumes: label-shape findings (correctness lens, or conventions for
the skill-auditor so labelForFinding reproduces the best-practice
variants; confidence defaulted to 0.7 per the production claims rule),
structured-schema lens findings validated as-is, and the validator's
{claims: [...]} verdicts into CaseVerification[]. It stages
claims.json for the validator, resolves {{#runtime-import}} directives
against the case tree (a case opts into a skills index by carrying the
file), retries once on malformed output with the rejection fed back,
keeps partial results when an agent fails twice, prefixes colliding
finding ids, and reports per-agent cost/turns/wall-clock.

live-runner.ts is the one module that touches a real model runtime:
an Agent SDK query() per dispatch with Read/Grep/Glob only, cwd pinned
to the staged checkout, the agent's pinned model, and hard turn and
wall-clock caps; plus the CLI smoke entry
(tsx live-runner.ts --case <id>, requires ANTHROPIC_API_KEY). The
investigation-cap CLI the prompts reference is not staged; its own
denied-budget fallback applies. Adds @anthropic-ai/claude-agent-sdk
as a dev dependency.

* [jwbron/live-eval-ab-runner] review: the live A/B runner (phase 3)

live-match.ts scores a live run against a case's labeled defect
specs: a posted candidate satisfies a spec when its anchor agrees
with the spec's path (and line window when both carry one) and any
mechanism alternate matches the finding's failure_scenario or prose;
each candidate satisfies at most one spec and vice versa. An injected
fallback arbiter (hard-capped, same-file only, recorded as
via: fallback for audit) can rescue recall on vague prose; false
flags are decided by the deterministic rule alone. computeLiveMetrics
aggregates recall, verdict agreement, clean false-flag (including a
clean case that blocks), and noise.

live-ab.ts is the arm orchestrator and CLI: baseline review.md from
git show <merge-base>, candidate from the working tree, both arms
over the same live corpus with everything else (corpus, lib, runner,
metrics, judge) from the candidate, per the plan's settled decision
to isolate the model seam. Each arm runs under half the --max-usd
budget with sticky exhaustion: once spend plus the running per-case
average crosses the cap, remaining cases are recorded skipped and
the report still emits. Spec-level regressions are diffed only over
cases both arms scored. Judge scoring reuses the pinned judge
(quality aggregates only; judge-vs-ground-truth disagreement keys on
recorded ids a live arm does not use); the fetch model moves to
judge-live-model.ts and live-judge.ts now imports it. runner.ts
gains an optional RunOptions.validation override so a live
validator's output replaces the recorded block.

Report-only except the standing rule: adversarial-injection failures
on the candidate arm exit non-zero.

* [jwbron/live-eval-ab-ci] review: per-PR live A/B workflow (phase 4)

Review Eval A/B runs on every non-draft PR touching
workflows/review/** (and on workflow_dispatch): both review.md arms
over the live corpus via live-ab.ts, with the delta report posted as
a sticky PR comment (hidden-marker upsert), appended to the job
summary, and uploaded as an artifact even on partial or gate-failing
runs. Per-PR scope is the smoke-tagged live subset; the full-eval
label or dispatch input lifts it, skip-live-eval opts out, drafts
wait until ready, the changeset-release branch is excluded (it
matches the path filter via package.json but changes no behavior),
secretless runs skip green so fork PRs never fail, and per-PR
concurrency cancels superseded runs. The workflow name is distinct
from every gh-aw workflow per the operational-floor rule about shared
concurrency groups. live-ab.ts gains --smoke-only and writes
out/live-ab-report.md alongside the JSON for the comment step.

* [jwbron/live-eval-corpus] review: exclude eval-corpus trees from lint and vitest

Tree directories are byte-exact case fixtures paired with each case's
diff: prettier auto-formatting one would silently desync it from the
diff the provenance gate parses, and a tree may carry a *.test.ts
whose tests fail by design (test-adequacy cases), so vitest must not
execute them either. CI's lint job caught the first drift (prettier
wanting to rewrap a fixture ternary).

* [jwbron/live-eval-producer-staging] review: namespace live finding ids with the case id

Live agents choose their own finding ids, so every case's first
correctness finding was live-correctness-reviewer-1; ids were unique
within a case but collided across cases, and judge.ts's score join
requires arm-global uniqueness. Caught by the first real A/B run
(both arms completed, then judge aggregation threw). Ids are now
<caseId>:<id> from the moment of parsing, so claims.json, the
validator round-trip, the matcher, and the judge all see the same
namespaced id.

* [jwbron/live-eval-ab-runner] review: judge failures degrade the A/B report instead of killing it

The first real A/B run spent both arms' budgets and then died in
judge aggregation, writing no report: the exact
everything-spent-nothing-posted failure mode the plan forbids. Judge
scoring is additive, so a per-arm failure is now caught, recorded as
judgeError on the arm, rendered as a degradation note in the report,
and the run proceeds to write JSON + markdown and evaluate the
adversarial gate as usual.

* [jwbron/review-trial-skill] review: add the review-trial skill (live A/B phase 5)

Packages the seeded-defect live-trial pattern (Khan/webapp#40678) as
a Claude Code skill so a trial costs an operator an afternoon instead
of a week of hand choreography. The skill collects required inputs
(seeded branch, human-authored defect table, arms, budget approval)
and refuses to improvise ground truth; sets up one isolated PR per
arm with per-arm trigger recipes and the distinct-workflow-name rule
(same-named gh-aw workflows share a per-PR concurrency group and
cancel each other); collects reviews, artifacts, and costs per run
with the known gh-aw artifact-bug tolerance; drives optional
lifecycle pushes; scores defect by defect with the deterministic
rule mirroring eval/live-match.ts plus audited manual judgment;
exports live-enabled corpus case skeletons with a sanitization gate
for private-repo content landing in this public repo; and cleans up
trial PRs, branches, and temporary workflows. Trials remain the
architecture-bet instrument; per-change evals belong to the corpus
A/B.

.gitignore narrows from .claude to .claude/* with a !.claude/skills/
carve-out: local agent state (settings, worktrees) stays untracked,
project skills are shared tooling and are committed.

* [jwbron/review-rereview-accountability] review: re-review accountability, code-rendered from the reconciler keep-list

* [jwbron/review-out-of-lane] review: hand off out-of-lane observations instead of dropping them

* [jwbron/review-out-artifact-upload] review: fix the out/ artifact upload (allowed-paths must match staging-relative paths)

* [jwbron/review-rereview-accountability] review: prettier-format the rereview module

* [jwbron/review-out-of-lane] review: prettier-format finding-schema

* [jwbron/review-dispatch-tax] review: dedupe lens discipline snippets into one staged disciplines file

* [jwbron/review-skill-rule-quote] review: render the quoted skill rule into the comment the author reads

* [jwbron/live-eval-corpus] review: route the specialist lens on each live case

The first live acceptance runs missed
incident-sql-missing-index:dm-missing-index-1 in all four arms: the
recorded finding is a data-migrations LENS finding, but live cases
carried no routerConfig lens rules, so the live roster never spawned
the specialist that catches it. Every live case whose recorded
finding belongs to a specialist lens now routes that lens on the
finding's file (seven cases across five lenses), mirroring how a
consumer ROUTING file would route the same paths in production.

* [jwbron/live-eval-ab-runner] review: carry agent-failure reasons into the A/B report

The acceptance runs surfaced a claim-validator failure that the
report could only name, not explain (perCase carried agent names
only, and the PerAgentReport.failed detail never reached the
markdown). perCase failedAgents entries are now '<agent>: <reason>',
so the next failure is diagnosable from the sticky comment alone.

* [jwbron/live-eval-ab-runner] review: close every A/B report with a per-row stability footer

The phase 4 acceptance pair measured which report rows a reader may
act on from one run: recall, verdict agreement, the regression lists,
and the adversarial gate reproduced exactly on the no-op control,
while judge quality and noise moved on jitter alone; and on the
weakened arm judge quality went UP 0.16 as recall fell 17 points
(fewer, surer comments each score better). The footer states this on
every report so nobody chases a judge-quality delta or reads one as
health.

* [jwies/review-budget-cap] review: clamp the run budget to the effective credit cap; land before the cap

The tier budget table assumes the workflow's $10 ceiling, but the
per-run credit cap is enforced runner-side and was invisible to the
agent, so a tighter consumer cap produced soft targets the hard cap
could not pay for: the 400-credit budget-shed behavior test planned
against the high tier's $10 targets, spent 390 credits on the finder
phase, and was killed by the api-proxy mid-persist with findings in
hand and nothing posted.

Three changes close the gap:

- The frontmatter sets max-ai-credits: 1000 explicitly and mirrors it
  into the agent environment as REVIEW_MAX_AI_CREDITS (consumers that
  override the cap keep the two in sync).
- The router resolves the effective cap (mirror, GH_AW_MAX_AI_CREDITS,
  a visible awf config, then the 1000-credit gh-aw default) and clamps
  the selected tier budget proportionally, floored at the trivial
  tier's values; routing.json carries capClamped and
  effectiveCreditCap so the orchestrator and post-hoc audits see the
  clamp.
- The budget prompt adds an estimated-credits proxy (in-band
  subagent_tokens sum / 5000) and two mandatory checkpoints, the
  critical one after the last finder returns and before claim
  validation, the exact boundary the observed run died at.

* review: split the credit-cap discovery and clamp into credit-cap.ts

router.ts crossed the 1000-line lint budget with the clamp addition; the
cap discovery (resolveCreditCap), the clamp (clampBudgetToCreditCap), and
the gh-aw default constant move to their own module by concern. The router
imports both and stays the single CLI entry point; tests for the moved
functions import from the new module. Prettier fix in router.test.ts rides
along.

* [jwbron/live-eval-ab-runner] review: identity short-circuit, gate-flip retry, drop-bucket taxonomy

Three instrument fixes from the eval-tuning memo, landed in the runner
so the whole stack above inherits them:

- Pre-flight identity short-circuit (memo item 1): byte-identical
  review.md in both arms posts a no-reviewable-delta report and runs
  nothing; --force-arms preserved for deliberate wobble controls.
- Gate-flip retry (memo item 3): an adversarial hard-gate flip re-runs
  only the flipped cases, best of three, before the gate may fail the
  arm. Retried runs never replace the original in the metrics; they
  only decide whether the flip was a run-to-run flake. Retry spend is
  recorded in the report.
- Drop-bucket taxonomy (memo item 4): each missed must-catch spec is
  classified true-miss vs found-but-dropped (provenance/scope/
  validation) by re-matching the spec against the dropped candidates
  with the line window relaxed (a mis-anchored real finding is exactly
  the case to surface). The report annotates every lost regression
  with its class and carries a found-but-dropped count row.

* [jwies/review-budget-cap] review: hold a landing reserve under the credit cap

The acceptance re-run showed the clamp engaging and the run still dying
at the hard cap (416/400 credits, mid-skill-audit): the clamped soft
target equaled the cap, spend is unobservable mid-run, and requests in
flight at the last checkpoint bill after it. The clamp now sets the
soft dollar target to 75% of the effective cap (LANDING_RESERVE_RATIO)
and scales the other targets to match, so shedding against the targets
lands the run inside the ceiling. The clamp also engages whenever the
tier's own target sits above the reserve-adjusted cap, not only above
the raw cap. review.md tells the orchestrator maxUsd is a landing
target, never money it may finish spending. Acceptance: re-run the
budget-shed test on the kept-alive webapp scaffold at cap 400.

* [jwbron/live-eval-ab-runner] review: judge economics (Haiku pin, retry with backoff)

The judge grades one comment's prose in 512 tokens; every load-bearing
metric (recall, verdict agreement, regressions, adversarial gate) is
deterministic and never touches it, and the acceptance runs showed the
judge signal is not single-run-stable on any model (it moved 0.11 on
the byte-identical control and went UP on the weakened arm). Price
that signal accordingly: the pin moves from claude-opus-4-8 to
claude-haiku-4-5-20251001 (a fifth of the cost; dated snapshot so
week-over-week scores compare; both arms always share one judge, so
within-run deltas are unaffected). Supersedes operator direction 4,
which pinned Opus. Also adds retry with backoff (3 attempts, 429/408/
5xx/network only) to the one live judge seam; a single transient 500
previously wasted an entire weekly scoring pass.

* [jwbron/live-eval-ab-ci] review: document why the baseline is the base tip, not the merge-base

The workflow passes origin/<base_ref> (the base branch tip) while its
comments and the dispatch-input description said merge-base. The code
is the right side of that disagreement: pull_request runs check out
the PR merge commit, so the candidate tree already contains the base
tip; baselining on the same tip isolates the PR's own review.md delta,
where a merge-base baseline would fold upstream review.md movement
into the PR's measured numbers. Prose updated to match the code; no
behavior change.

* [jwbron/live-eval-ab-runner] review: type the judge response via the fetch signature (eslint no-undef)

* [jwies/review-budget-cap] review: rename the landing ratio to what it holds; test the cap-resolution guards

* [jwies/review-budget-cap] review: split the credit-cap tests into credit-cap.test.ts (router.test.ts hit the max-lines lint limit)

* [jwies/review-budget-cap] review: split the budget table into budgets.ts (router.ts over the max-lines limit after the main merge)